HDMI Capture Problem SOLVED - AviSynth RULES!

R

Ralph B

New member
After much work, I have succeeded in extracting a clean 24P from GH2 HDMI captures.


First, a recap of the problem:

The GH2 sensor scans at 23.976P and embeds the picture in a 59.94i stream by adding 3:2 pulldown. Nothing unusual about that. But, when it sends this signal out the HDMI port, it adds in extra fields to make the signal a true 60.00i. These extra duplicate fields create havoc when attempting to remove the 3:2 pulldown. Plus, they throw the audio increasingly out of sync the longer the camera rolls.

But AviSynth comes to our rescue! The following script works for both NTSC and PAL cameras.

NOTE: This is an advanced script posted on April 24, 2011, and it superceeds the original script.

ADVANCED SCRIPT TO PROCESS 1080 24P FOOTAGE


Code: 
avisource("Your HDMI movie.avi")

LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\TIVTC.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\FDecimate.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\mvtools2.dll")

function filldrops (clip c)
{
**even = c
**super_even=MSuper(even,pel=2)
**vfe=manalyse(super_even,truemotion=true,isb=false,delta=1)
**vbe=manalyse(super_even,truemotion=true,isb=true,delta=1)
**filldrops_e = mflowinter(even,super_even,vbe,vfe,time=50)

**ConditionalFilter(even, filldrops_e, even, "YDifferenceFromPrevious()", "lessthan", "1.3")

}


# 3:2 Pulldown Removal
AssumeTFF()
trim(1,0)
TFM(mchroma=false)
FDecimate(threshold=0.5)
AssumeFPS(24000,1001)

# Chroma Fix
FixBrokenChromaUpsampling() 
converttoyv12()

# Fix Residual Duplicates
filldrops()

#Audio Delay
delayaudio(.07)

# Show the Metric for filldrops
# This is a diagnostic to determine the threshold "lessthan" in the ConditionalFilter
# To run, remove the # in the line below and put a # to the left of filldrops() 7 lines above
# ScriptClip(last,"Subtitle(String(YDifferenceToNext() ))" )


WHAT YOU WILL NEED

In order to run this script you need to install Avisynth and have TIVTC.dll, FDecimate.dll and mvtools2.dll inside your Avisynth plugins folder.

There are many versions of Avisynth available. Alpha 2.6.0 has been tested with Windows XP and works:
http://sourceforge.net/projects/avi.../AviSynth_090927.exe/download?use_mirror=iweb

However, I had some problems using it with Windows 7. Maybe it was just my machine. So, for Windows 7, I ended up using Avisynth 2.5.8. YMMV.

You can get TIVTC here.
http://bengal.missouri.edu/~kes25c/
It's about 3/4 the way down the page - TIVTC - 01/17/2008 v1.0.5

fdecimate is here:
http://neuron2.net/fdecimate/fdecimate.html

mvtools2 is here:
http://avisynth.org.ru/mvtools/mvtools2.html

You will also need a program to display the output from Avisynth and to render the processed footage. VirtualDub works fine.
http://virtualdub.sourceforge.net/
I recommend the 32-bit version.

**************************************************

PROCEDURE

One very important thing I must point out... you must set your capture device to record at 30fps, not 29.97. This is because the camera really is putting out 30fps. If you record at 29.97, the script won't work properly.

You also must record to a 4.2.2 codec. The script will not work well if you record to a 4.2.0 codec.
The following codecs have been successfully tested: UT 422 codec, Cineform, Blackmagic MJPEG codec.

I also recommend rolling the camera first, then starting the external capture. And stop the external capture before stopping the camera. This will give you the cleanest signal and avoid the discontinuity that occurs when the GH2 switches into and out of record.

When running the script, replace the words "Your HDMI movie.avi" with the actual name of the movie clip + it's file extension. Be sure quotes surround the name.

The GH2 is truly a phenomenal camera. And the ability to record an uncompressed 24P HDMI signal makes it even more phenomenal!

**************************************************

SCRIPT NOTES


AUDIO with HDMI

The GH2 does not send audio through the HDMI, only picture. If your HDMI capture device has analog audio inputs, you can capture audio along with the picture, which is a desirable thing to do. However, there is a problem - the GH2 introduces a delay to the picture when it sends it out the HDMI, so the audio and picture will not be in sync. In the script, "delayaudio(.07)" fixes this. It delays the audio the precise amount to match the picture. The number .07 works with the Blackmagic Intensity board. Other capture devices may or may not require a different setting. So, be sure to test.



BATCH PROCESSING

If you use the script in a production environment, with hundreds and maybe thousands of clips, you're going to want to automate the process. Fortunately, someone has written a batch processing program for Avisynth scripts. Get it here:

http://www.hv20.com/showthread.php?25044-Batch-Intermediate-Creation-Utilities/page4

Scroll down the page to post #94 and download the 'Cosmetic Update to v1.6'.

There are actually four programs in the package, three of which basically do the same thing. I found VDub-BIC to be the most stable, and it's the one I use to batch process the scripts.



WORKING with ProRes

If your HDMI capture device records to ProRes (such as, the Atomos Ninja), you will need to modify the script and install additional software. In the very first line, replace "avisource" with "qtinput".

Download and install Quicktime and Apple's ProRes decoder for Windows.

Quicktime
http://support.apple.com/kb/dl837

Prores Decoder
http://support.apple.com/downloads/Apple_ProRes_QuickTime_Decoder_1_0_for_Windows

Next, download QTsource. This will enable Avisynth to directly open Quicktime movies.

http://www.tateu.net/software/

Put qtsource.dll in your Avisynth plugins folder, and you're good to go.



CHROMA FIX

When recording through the HDMI, the GH2 exhibits chroma jaggies in the red channel. Fortunately, Avisynth can correct this. These pictures are enlarged 2X for clarity. Thanks to GrgurMG for helping with the development of this section of the script.

Original.jpg
Original

Chroma Fix.jpg
with Chroma Fix

The Advanced Script has a streamlined Chroma Fix that renders twice as fast as the original Chroma Fix.
The difference between the two Chroma Fixes is virtually imperceptible.

However, if you would like to run the original Chroma Fix, just replace the Chroma Fix section of the script with this:

Code: 
#Original Chroma Fix
**FixBrokenChromaUpsampling() 
# Seperates the YUV channels. YV24 seems neccessary for the split and remerge to work.
**ConvertToYV24(chromaresample="spline36")
**Y = Tweak(sat=0, coring=false)
**U = UToY()
**V = VToY()
# Shifts the YUV channels individually. In this case subpixel(halfpixel) shifting.
**U = U.Spline36Resize(u.width,u.height,0.5,-0.5)
**V = V.Spline36Resize(v.width,v.height,0,-0.5)
**Y = Y.Spline36Resize(y.width,y.height,0,0.5)
# Manually squeezes the chroma down to 4:2:0 and back. Prevents further shifting.
**U = U.Spline36Resize(u.width,u.height/2)
**V = V.Spline36Resize(v.width,V.height/2)
**U = U.Spline36Resize(u.width,u.height*2)
**V = V.Spline36Resize(v.width,v.height*2)
# Recombines individual YUV channels.
**YToUV(U,V,Y)
# Depending on the source file you may need to convert from YV24 back to YUY2/YV12.


IS THE HDMI OUTPUT 4.2.2 ?

The question of whether the HDMI output is 4.2.2 constantly comes up. The fact is, nobody knows because the chroma is so mangled up. What we know for certain is that after applying the Chroma Fix, we have a clean, uncompressed 4.2.0 picture. As a practical matter, this is the best way to think of it.

*************************************************
HBR mode with FIRMWARE 1.1 and HDMI

Firmware version 1.1 adds HBR mode, which is frame rate 29.97P wrapped in a 59.94i stream. For PAL users it's 25P wrapped in a 50i stream. How does this signal come through the HDMI port? Here's the scoop: 29.97P is converted to 30P and is a disaster, hopelessly mangled up. Extra fields are added in, just like for 24P. However, there's good news for PAL users - 25P comes through cleanly - no extra fields. There is still the issue of the jaggy chroma in the red channel, even in 25P. So if you want a perfect picture, you would still need to run it through Avisynth. The script would consist of only the chroma fix. I suspect most people won't bother. It's only really noticable in areas of heavily saturated red. The one exception is if you're doing greenscreen keying with the HDMI picture. Then you must use the chroma fix because the red jaggies affect the edges of flesh. Without the chroma fix, it's going to be very hard to pull a good key.

*************************************************
A WORD OF ENCOURAGEMENT

I realize all of this can sound overwhelming, especially for someone who has no experience using Avisynth. But if you're willing to put in the time to get over the initial learning hump, the actual act of processing your clips is a no brainer - just load the files, push a button and go. Also, learning how to use Avisynth is a valuable skill in it's own right. There's a wealth of scripts available for every conceivable (and sometimes inconceivable) function for processing video.

For example, check this page out:

http://avisynth.org/mediawiki/External_filters

*************************************************
FINAL NOTE

If you're coming here for the first time, be aware that you don't have to read through this lengthy thread. All the important information is right here in the first post. I have updated this post many times to incorporate the latest findings. If you do read through the thread, you're actually going back in time.

*************************************************
LOW LIGHT TRICKS for the GH2

Here's a useful link:
http://www.dvxuser.com/V6/showthread.php?237584-HDMI-Capture-Problem-SOLVED-AviSynth-RULES!/page55

*************************************************
*************************************************
Technical Documentation

This section is for technical troubleshooting. Reading it is optional. You can skip this section and use the script just fine.

TESTING PROCEDURE

My testing procedure is to record a three minute clip with a head slate and a tail slate.
I place the MTS clip (that was recorded in camera) on the timeline in an editing program. This becomes my reference. Then I place the Avisynth processed clip on a track immediately above the MTS clip. Align the clips so the head slates match. Go to the end and see whether the tail slates match. Next, change the opacity of the Avisynth clip to 50%, which enables both clips to be seen at once. Then step through the timeline to make sure each frame is correctly aligned.

ABOUT THE THRESHOLDS

The advanced script has two thresholds in it. You probably won't have to adjust them, but for the sake of completeness, here is their documentation.

The first threshold is in fdecimate and it is important for the correct operation of the script. If you notice skipped frames or duplicate frames, the threshold might need adjustment. If the threshold is set too low, there will be duplicate frames. If the threshold is set too high, there will be skipped frames. The way to find the correct threshold is to turn on Metrics in fdecimate. In the script, temporarily replace fdecimate(threshold=whatever) with fdecimate(metrics=true). This will give you a numeric readout of the difference between each frame. The objective is to get rid of the duplicate frames. Run the script. As you step through the clip, make a note of the numbers associated with the duplicate frames. You want to set the threshold slightly higher than the largest of these numbers. In the script, replace fdecimate(metrics=true) with fdecimate(threshold=the number you found).

I found that a threshold of 0.5 works well if the footage has been recorded to a 4.2.2 codec at high quality.

You probably won't have to adjust the threshold, but if you do, be sure to run metrics first, so you have "hard numbers" to work with.

The second threshold is in the ConditionalFilter. It is the number to the right of "lessthan". If any duplicate frames manage to slip through fdecimate, the ConditionalFilter will catch them and interpolate inbetween frames usings mvtools2.

If you have any duplicate frames in the final output, I would examine this threshold before looking at the threshold in fdecimate, since this is where the problem is most likely to be. This threshold has a metric you can turn on to get a numerical readout of the difference brtween frames. The instructions are in the last four lines of the script. This threshold is currently set to 1.3. Duplicate frames should have a number less than the threshold and frames that are different should have a number greater than the threshold.

**************************************************
**************************************************

SCRIPT to PROCESS 1080 60i FOOTAGE

Here's a script designed to remove the extra fields from 1080 60i clips that have been captured through the HDMI.

This script is not perfect. The processed clip will occassionally fall one field behing the in-camera clip, and then catch up. I'm posting it for people who want to play around with it, but I would not recommend shooting "serious" footage with it.


Code: 
aviSource("Your 1080 60i HDMI movie")

Load_Stdcall_plugin("C:\Program Files\AviSynth 2.5\plugins\yadif.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\FDecimate.dll")

Yadif(mode=1, order=1)
trim(1,0)

A=converttoyv12()

skip = 0 # no of frames skipped
pf = 0 # previous frame
A.Greyscale().FDecimate(rate=59.94, threshold=2.2)
ScriptClip("""
**GScript("
** *skip = current_frame < pf ? 0 : max(skip-2, 0)
** *while (current_frame+skip < Framecount(A) && LumaDifference(A.Trim(skip, 0)) > 0) {
** * *skip = skip+1 # skip decimated frame
** *}
**")
**pf = current_frame
**return A.Trim(skip, 0)
""")

delayaudio(.07)

AssumeFrameBased()
AssumeTFF()
SeparateFields()
SelectEvery(4, 0, 3)
Weave()
AssumeFPS(30000,1001)


In order to run this script, you need to have fdecimate.dll, yadif.dll and gscript.dll inside your Avisynth plugin folder.

You can download Yadif here:
http://avisynth.org.ru/yadif/yadif.html

Get Gscript here:
http://forum.doom9.org/showthread.php?t=147846
It's at the end of the first post - Attached Files
GScript_11.zip

fdecimate is here:
http://neuron2.net/fdecimate/fdecimate.html


**************************************************

SCRIPT NOTE 1: This script works by converting the footage into a 1080 60P stream in order to remove the duplicate fields. It then re-interlaces it at the end. However, there may be situations where it's desirable to not re-interlace and to output the 1080 60P stream. Such scenarios might include pulling blue or greenscreen keys, adding in computer graphics, motion tracking, and compositing in general.

If you wish to output the 60P stream, just delete this last section of the script:

AssumeFrameBased()
AssumeTFF()
SeparateFields()
SelectEvery(4, 0, 3)
Weave()
AssumeFPS(30000,1001)


SCRIPT NOTE 2: As with the 24P script, if you're recording audio through your HDMI device, be sure to run your own tests to determine the proper audio delay.

**************************************************
MY OWN CONTRIBUTION TO THE HACK

In addition to the Avisynth scripts, I've also created my own settings for the GH2 hack, which I call 'SANITY'.
Check it out here:

http://www.dvxuser.com/V6/showthread.php?316678-SANITY-X-High-Bitrate-Versiion-of-Sanity
 
Last edited:
I had written off HDMI capture because I read it was useless. I guess now I'll have to do a bunch of reading on which capture device to consider. Thanks, Ralph B! This is going to excite a lot of people when they wake up.
 
are you sure you're able to capture a better output than MTS? that's the main thing to check (if you could post FF jpegs?)
 
It wasn't deemed useless because of the lack of consistency for removing pulldown. It was deemed useless because the color gets trashed coming out of the HDMI pipe, somehow. The AVISynth solution is elegant, but doesn't solve why one would want uncompressed video that looks like this http://www.dvxuser.com/V6/showthread.php?231315-Any-GH2-news-on-the-HDMI-output-front/page27
While being "uncompressed" a close look at the HDMI image makes you realize you're not getting 4:2:2 resolution. Which kinda defeats the purpose of recording the image "uncompressed" to start with.
At least that's where that conversation died.
 
I've been doing film and video work professional for about 8 years and I've never once heard of anyone trying to record the picture from an HDMI output from a live video feed. HDMI monitor out is always just used for a reference when shooting. So the director and/or dp can view the shot and make lighting adjustments, etc.

That's great that you were able to get a full 24p out of the monitor now, but it really would have been fine without.
 
80FOURENT said:
I've been doing film and video work professional for about 8 years and I've never once heard of anyone trying to record the picture from an HDMI output from a live video feed. HDMI monitor out is always just used for a reference when shooting. So the director and/or dp can view the shot and make lighting adjustments, etc.
Click to expand...

Have you been hiding under a rock for the last 8 years? Ever heard of a Nanoflash, Ki Pro, Ki Pro mini, Atomos Ninja, IO HD, and pretty much every other product AJA makes.

80FOURENT said:
That's great that you were able to get a full 24p out of the monitor now, but it really would have been fine without.
Click to expand...

Maybe to you, but some of us are on a hunt for better quality.
 
Last edited:
No, so far signs have shown us that we don't get 4:2:2 out of HDMI which is one of the main reasons why there is much doubt as to whether it's worth it. but Ralph B has made the first big step, so now we need to start doing comparisons to see what improves by capturing the HDMI.
 
Ralph B, I think this is a good sign. I'm not sure why some folks can't see this as a good solution (even if it turns out not to be 4:2:2). The benefit is...if you don't want AVCHD...you now have a choice. I saw some HDMI footage early on and I thought it looked a tad sharper than the AVCHD. I could be wrong but that was my first impression. It might not be worth it to some but it would be worth it to those why already have a HDMI capture solution. I say...if this works correctly...then it's worth it to get one of those capture solutions....especially knowing that better cameras come out all of the time and you could always upgrade to the next best thing...and STILL have you capture device.
 
Ralph, how does the script know where the spurious duplicate fields are? I mean, I would assume they're distributed evenly throughout, but -- how does it know where the first one is? It could be at any position, depending on when the user presses "record"...

If someone wants to play with this, here's a file recorded from the GH2 onto the NanoFlash, shooting a consistent moving object at an identical speed, so it should be easy to detect any duplicated fields...
http://www.mediafire.com/?ib3arayi6cw1ic3
 
Sandbox said:
It wasn't deemed useless because of the lack of consistency for removing pulldown. It was deemed useless because the color gets trashed coming out of the HDMI pipe, somehow. The AVISynth solution is elegant, but doesn't solve why one would want uncompressed video that looks like this http://www.dvxuser.com/V6/showthread.php?231315-Any-GH2-news-on-the-HDMI-output-front/page27
While being "uncompressed" a close look at the HDMI image makes you realize you're not getting 4:2:2 resolution. Which kinda defeats the purpose of recording the image "uncompressed" to start with.
At least that's where that conversation died.
Click to expand...

Thanks for the input. I've updated the script. Check out the first post.
 
Barry_Green said:
Ralph, how does the script know where the spurious duplicate fields are? I mean, I would assume they're distributed evenly throughout, but -- how does it know where the first one is? It could be at any position, depending on when the user presses "record"...
Click to expand...

The magic key is the TFM function, Barry. It has the ability to sift through combed frames and extract progressive pictures wherever they happen to be. It was written by Tritical, who is one of the Avisynth gurus. TFM deletes nothing. So, you're left with a stream of progessive fames with a lot of duplicates scattered throughout. Now the task is to get rid of the duplicates, and that's what fdecimate does. Then we set the frame rate to 23.976 and viola, we're home.

Of course, the script is only as good as the underlying functions, and it's entirely conceivable that a certain piece of footage might trip them up. But, so far they've performed fine.

We do need people putting the script to the test to see how robust it really is.
 
Last edited:
@Ralphi B - Nice workaround with the ConvertToYV12() !

If I can make some suggestions:

1) I would use AssumeFPS(24000,1001) instead of AssumeFPS(23.976) . Gradual desync will occur with the latter on longer clips, because of the approximation

2) DirectShowSource() isn't necessarily frame accurate, and relies on system installed codecs/filters - this can cause inconsistencies. This may be problematic for some people, especially if you are doing non linear seeks or using temporal filters. If you have an AVI wrapped file I would recommend using AVISource(). If you have a MOV wrapped file (maybe from a AJA capture), I would use FFMpegSource or QTInput()

TIVTC includes the function TDecimate() which is fairly similar to FDecimate(). Is there a reason why you prefer FDecimate() ?



Barry_Green said:
Ralph, how does the script know where the spurious duplicate fields are? I mean, I would assume they're distributed evenly throughout, but -- how does it know where the first one is? It could be at any position, depending on when the user presses "record"...

If someone wants to play with this, here's a file recorded from the GH2 onto the NanoFlash, shooting a consistent moving object at an identical speed, so it should be easy to detect any duplicated fields...
http://www.mediafire.com/?ib3arayi6cw1ic3
Click to expand...



If you're using this for nanoflash footage, you can read out of mxf container with ffmpegsource2
http://code.google.com/p/ffmpegsource/

Another alternative is to use DGIndex and MPEG2Source() , but this requires unwrapping the .m2v elementary video from the .mxf container.

RalphieB's default settings won't work on your clip, because there is field switching in your sample and longer strings of duplicates , so both TFM and TDecimate must use different parameters

FFVideoSource("01293001.MXF",seekmode=0)
AssumeTFF()
TFM(order=1,mode=7,slow=2)
TDecimate(mode=1)
AssumeFPS(24000,1001)
ConvertToYV12() #If you want to convert to 4:2:0

(I've scaled this down by 2x, as it's just for demonstration purposes for the cadence of Barry's clip. It's MJPEG in AVI, if anyone needs a free decoder , install FFDSHOW. The VFW decoder configuration needs to have MJPEG enabled. You can examine frame by frame in virtualdub for example)
http://www.mediafire.com/?2cl8m8agohp63lu

(In theory, those parameters should work for other GH2 HDMI clips as well, but more testing of course is better)
 
Last edited:
You must log in or register to reply here.
Back
Top